Implement the wazuh-server daemon initialization

GGP1 commented 1 month ago

Description

In Wazuh v5.0, the wazuh-server daemon will be in charge of initiating all the other daemons.

This issue aims to implement the logic necessary to fulfill the functional and non-functional requirements listed below.

wazuh-server structure diagram

![wazuh-server](https://github.com/user-attachments/assets/a55af5f3-59c3-4ed7-a38c-fbb5d1c591e2)

Functional requirements

wazuh-server must be always running and will be the principal daemon of the Wazuh Server, initializing first before any other daemon that it will start via the Process Control task.
The wazuh-server daemon must be able to capture the sub-processes output and instruct them whether to log messages to a file, remote host or just the standard streams.

Non-functional requirements

The cluster default configuration must be changed following the new capacities in the upcoming configuration template.

Implementation restrictions

The current socket used for the communications must be the one available for the tasks to be carried out by the cluster.

Tasks

[x] Develop the code to initiate the Wazuh Server.
[x] Validate that the cluster starts all other daemons and processes successfully.

GGP1 commented 1 month ago

Update

After making the modifications to use a cluster and to listen on localhost by default, I realized that the pull request was becoming too big and decided to split the work into two different issues.

I opened https://github.com/wazuh/wazuh/issues/25895 to continue working on that and I kept this one focused on initializing the wazuh-server and all the subprocesses.

GGP1 commented 1 month ago

Update

Started by creating a docker image without all the legacy modules and that uses wazuh_clusterd.py as the entrypoint. I'm still working on installing the embedded python interpreter without unnecessary dependencies.

GGP1 commented 1 month ago

Update

I tried setting up an environment with the embedded and then installing the dependencies in the interpreter but both require several days of work to get 5.0 working without the current installation.

I continued with the initialization in the current environment and making changes to initialize both APIs during cluster startup.

GGP1 commented 1 month ago

Update

Implemented part of the initialization, tested the changes and updated unit tests. To finish the development, I must contact the CppServer team to ask for guidance to execute the Engine and do some extra tests.

GGP1 commented 1 month ago

Update

Added the engine process initialization and validated that everything runs as expected after the changes.

GGP1 commented 1 month ago

Update

I simplified the way in which we started the daemons, included the wazuh-engined PID file in the folder where the others are stored so the manager status takes it into account.

We decided not to include changes to the naming and paths in this PR, since that would require changing thousands lines of code. We will be performing those changes in new issues.

New tests

Start the cluster in foreground mode

```console root@wazuh-master:/# /var/ossec/bin/wazuh-clusterd -rf Starting cluster in foreground (pid: 101021) 2024/10/09 14:34:12 INFO: [Cluster] [Main] Started wazuh-engined (pid: 101022) 2024/10/09 14:34:12 INFO: [Cluster] [Main] Started wazuh-apid (pid: 101023) 2024/10/09 14:34:12 INFO: [Cluster] [Main] Started wazuh-comms-apid (pid: 101024) 2024-10-09 14:34:12.265 101022:101022 info: Logging initialized. 2024-10-09 14:34:12.265 101022:101022 info: Store initialized. 2024-10-09 14:34:12.265 101022:101022 info: RBAC initialized. 2024-10-09 14:34:12.265 101022:101022 info: MetricsManager: Created new scope: (KVDB) 2024-10-09 14:34:12.283 101022:101022 info: KVDB initialized. 2024-10-09 14:34:12.283 101022:101022 info: Geo initialized. 2024-10-09 14:34:12.294 101022:101022 info: Schema initialized. 2024/10/09 14:34:12 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock 2024/10/09 14:34:12 INFO: [Master] [Main] Serving on ('0.0.0.0', 1516) 2024/10/09 14:34:12 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:34:12 INFO: [Master] [Local integrity] Finished in 0.099s. Calculated metadata of 3 files. 2024/10/09 14:34:12 INFO: Starting API in foreground 2024/10/09 14:34:12 INFO: Starting API as root 2024/10/09 14:34:12 INFO: Checking RBAC database integrity... 2024/10/09 14:34:12 INFO: /var/ossec/api/configuration/security/rbac.db file was detected 2024/10/09 14:34:12 INFO: RBAC database integrity check finished successfully 2024-10-09 14:34:13.037 101022:101022 info: Loaded timezone database version: '2024a' 2024-10-09 14:34:13.040 101022:101022 info: HLP initialized. 2024-10-09 14:34:13.062 101022:101022 info: Builder initialized. 2024-10-09 14:34:13.062 101022:101022 info: Catalog initialized. 2024-10-09 14:34:13.062 101022:101022 info: Policy manager initialized. 2024-10-09 14:34:13.062 101022:101022 info: MetricsManager: Created new scope: (EventQueue) 2024-10-09 14:34:13.062 101022:101022 info: MetricsManager: Created new scope: (EventQueueDelta) 2024-10-09 14:34:13.063 101022:101022 info: The queue will be flooded in the file: /var/ossec/logs/engine-flood.log 2024-10-09 14:34:13.064 101022:101022 info: MetricsManager: Created new scope: (TestQueue) 2024-10-09 14:34:13.064 101022:101022 info: MetricsManager: Created new scope: (TestQueueDelta) 2024-10-09 14:34:13.069 101022:101022 info: No flooding file provided, the queue will not be flooded. 2024-10-09 14:34:13.069 101022:101022 warning: Router: router/router/0 table is empty 2024-10-09 14:34:13.069 101022:101022 warning: Router: router/tester/0 table is empty 2024-10-09 14:34:13.069 101022:101022 info: Router initialized. 2024-10-09 14:34:13.069 101022:101022 info: Starting database file decompression. 2024-10-09 14:34:13.091 101022:101022 error: Error opening the database: Couldn't find column family: 'vendor_map', trying to re-download the feed. 2024-10-09 14:34:13.091 101022:101022 info: MetricsManager: Created new scope: (endpointAPI) 2024-10-09 14:34:13.091 101022:101022 info: MetricsManager: Created new scope: (endpointAPIRate) 2024-10-09 14:34:13.091 101022:101022 info: MetricsManager: Created new scope: (endpointEvent) 2024-10-09 14:34:13.091 101022:101022 info: MetricsManager: Created new scope: (endpointEventRate) 2024-10-09 14:34:13.093 101022:101022 info: Starting the server... 2024/10/09 14:34:13 INFO: Starting API in foreground 2024/10/09 14:34:13 INFO: Starting API as root 2024/10/09 14:34:13 INFO: Listening on 0.0.0.0:27000 2024/10/09 14:34:13 INFO: Starting gunicorn 22.0.0 2024/10/09 14:34:13 INFO: Listening at: https://0.0.0.0:27000 (101024) 2024/10/09 14:34:13 INFO: Using worker: uvicorn.workers.UvicornWorker 2024/10/09 14:34:13 INFO: Booting worker with pid: 101163 2024/10/09 14:34:13 INFO: Started server process [101163] 2024/10/09 14:34:13 INFO: Waiting for application startup. 2024/10/09 14:34:13 INFO: Booting worker with pid: 101170 2024/10/09 14:34:13 INFO: Application startup complete. 2024/10/09 14:34:13 INFO: Booting worker with pid: 101171 2024/10/09 14:34:13 INFO: Started server process [101170] 2024/10/09 14:34:13 INFO: Waiting for application startup. 2024/10/09 14:34:13 INFO: Application startup complete. 2024/10/09 14:34:13 INFO: Started server process [101171] 2024/10/09 14:34:13 INFO: Waiting for application startup. 2024/10/09 14:34:13 INFO: Application startup complete. 2024/10/09 14:34:13 INFO: Booting worker with pid: 101213 2024/10/09 14:34:13 INFO: Started server process [101213] 2024/10/09 14:34:13 INFO: Waiting for application startup. 2024/10/09 14:34:13 INFO: Application startup complete. 2024/10/09 14:34:15 INFO: Listening on 0.0.0.0:55000. 2024/10/09 14:34:15 INFO: Getting installation UID... 2024/10/09 14:34:15 INFO: Getting updates information... 2024/10/09 14:34:20 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:34:20 INFO: [Master] [Local integrity] Finished in 0.003s. Calculated metadata of 3 files. ```

List processes

```console root@wazuh-master:/# ps -ef --forest UID PID PPID C STIME TTY TIME CMD root 13158 0 0 12:30 pts/1 00:00:00 bash root 111373 13158 0 14:43 pts/1 00:00:00 \_ ps -ef --forest root 53 0 0 12:04 pts/0 00:00:00 bash root 101013 53 0 14:34 pts/0 00:00:00 \_ /bin/sh /var/ossec/bin/wazuh-clusterd -rf root 101021 101013 0 14:34 pts/0 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/framework/scripts/wazuh_clusterd.py -rf root 101022 101021 0 14:34 pts/0 00:00:00 \_ /var/ossec/bin/wazuh-engine server start root 101023 101021 0 14:34 pts/0 00:00:03 \_ /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r -f root 101068 101023 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r -f root 101071 101023 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r -f root 101074 101023 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r -f root 101024 101021 0 14:34 pts/0 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101138 101024 2 14:34 pts/0 00:00:15 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101155 101024 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101157 101024 1 14:34 pts/0 00:00:10 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101163 101024 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101170 101024 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101171 101024 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101213 101024 0 14:34 pts/0 00:00:00 | \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r -f root 101064 101021 0 14:34 pts/0 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/framework/scripts/wazuh_clusterd.py -rf root 101065 101021 0 14:34 pts/0 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/framework/scripts/wazuh_clusterd.py -rf root 1 0 0 12:03 ? 00:00:00 bash /scripts/entrypoint.sh wazuh-master master-node master root 111207 1 0 14:43 ? 00:00:00 sleep 10 ```

Shutdown

```console 2024/10/09 14:43:41 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:43:41 INFO: [Master] [Local integrity] Finished in 0.001s. Calculated metadata of 3 files. ^C2024-10-09 14:43:48.407 101022:101022 info: Stopping the server 2024/10/09 14:43:48 INFO: Handling signal: int 2024-10-09 14:43:48.407 101022:101022 info: Server closed 2024-10-09 14:43:48.407 101022:101022 info: [Endpoint: /var/ossec/queue/sockets/queue] Closed. 2024-10-09 14:43:48.407 101022:101022 info: [Endpoint: /var/ossec/queue/sockets/engine-api] Closed 2024-10-09 14:43:48.407 101022:101022 info: Server stopped 2024-10-09 14:43:48.407 101022:101022 info: API terminated. 2024/10/09 14:43:48 INFO: [Cluster] [Main] SIGINT received. Shutting down... 2024/10/09 14:43:48 INFO: [Cluster] [Main] Shutting down wazuh-engined (pid: 101022) 2024/10/09 14:43:48 INFO: [Cluster] [Main] Shutting down wazuh-apid (pid: 101023) 2024/10/09 14:43:48 INFO: [Cluster] [Main] Shutting down wazuh-comms-apid (pid: 101024) 2024/10/09 14:43:48 INFO: Shutting down root@wazuh-master:/# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 7372 1928 ? Ss 12:03 0:00 bash /scripts/entrypoint.sh wazuh-master master-node master root 53 0.0 0.0 7636 3024 pts/0 Ss 12:04 0:00 bash root 13158 0.0 0.0 7636 3852 pts/1 Ss+ 12:30 0:00 bash root 111703 0.0 0.0 5772 1060 ? S 14:43 0:00 sleep 10 root 111744 0.0 0.0 10072 1532 pts/0 R+ 14:44 0:00 ps aux ```

Start the cluster in background mode

```console root@wazuh-master:/# /var/ossec/bin/wazuh-clusterd -r root@wazuh-master:/# ps -ef --forest UID PID PPID C STIME TTY TIME CMD root 13158 0 0 12:30 pts/1 00:00:00 bash root 53 0 0 12:04 pts/0 00:00:00 bash root 112259 53 0 14:44 pts/0 00:00:00 \_ ps -ef --forest root 1 0 0 12:03 ? 00:00:00 bash /scripts/entrypoint.sh wazuh-master master-node master root 112029 1 0 14:44 ? 00:00:00 sleep 10 root 112040 1 8 14:44 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/framework/scripts/wazuh_clusterd.py -r root 112041 112040 1 14:44 ? 00:00:00 \_ /var/ossec/bin/wazuh-engine server start root 112196 112040 0 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/framework/scripts/wazuh_clusterd.py -r root 112197 112040 0 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/framework/scripts/wazuh_clusterd.py -r root 112083 1 75 14:44 ? 00:00:02 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r root 112084 112083 0 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r root 112087 112083 0 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r root 112090 112083 0 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py -r root 112157 1 2 14:44 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root 112158 112157 3 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root 112175 112157 0 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root 112177 112157 1 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root 112183 112157 2 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root 112184 112157 1 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root 112185 112157 2 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root 112186 112157 1 14:44 ? 00:00:00 \_ /var/ossec/framework/python/bin/python3 /var/ossec/apis/scripts/wazuh_comms_apid.py -r root@wazuh-master:/# ls -la /var/ossec/var/run/ total 48 drwxrwx--- 1 root wazuh 4096 Oct 9 14:44 . drwxr-x--- 1 root wazuh 4096 Oct 9 14:44 .. -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-apid-112083.pid -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-apid_auth-112087.pid -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-apid_events-112090.pid -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-apid_exec-112084.pid -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-clusterd-112040.pid -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-clusterd_child_0-112196.pid -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-clusterd_child_1-112197.pid -rw-r--r-- 1 root root 7 Oct 9 14:44 wazuh-comms-apid-112157.pid -rw-r----- 1 root root 7 Oct 9 14:44 wazuh-engined-112041.pid ```

Shutdown

```console root@wazuh-master:/# kill 112040 root@wazuh-master:/# ps -ef --forest UID PID PPID C STIME TTY TIME CMD root 13158 0 0 12:30 pts/1 00:00:00 bash root 53 0 0 12:04 pts/0 00:00:00 bash root 112903 53 0 14:45 pts/0 00:00:00 \_ ps -ef --forest root 1 0 0 12:03 ? 00:00:00 bash /scripts/entrypoint.sh wazuh-master master-node master root 112901 1 0 14:45 ? 00:00:00 sleep 10 ```

cluster.log

```console 2024/10/09 14:44:35 INFO: [Cluster] [Main] Started wazuh-engined (pid: 112041) 2024/10/09 14:44:36 INFO: [Cluster] [Main] Started wazuh-apid (pid: 112083) 2024/10/09 14:44:37 INFO: [Cluster] [Main] Started wazuh-comms-apid (pid: 112157) 2024/10/09 14:44:37 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock 2024/10/09 14:44:37 INFO: [Master] [Main] Serving on ('0.0.0.0', 1516) 2024/10/09 14:44:37 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:44:37 INFO: [Master] [Local integrity] Finished in 0.099s. Calculated metadata of 3 files. 2024/10/09 14:44:45 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:44:45 INFO: [Master] [Local integrity] Finished in 0.001s. Calculated metadata of 3 files. 2024/10/09 14:44:53 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:44:53 INFO: [Master] [Local integrity] Finished in 0.001s. Calculated metadata of 3 files. 2024/10/09 14:45:01 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:45:01 INFO: [Master] [Local integrity] Finished in 0.001s. Calculated metadata of 3 files. 2024/10/09 14:45:09 INFO: [Master] [Local integrity] Starting. 2024/10/09 14:45:09 INFO: [Master] [Local integrity] Finished in 0.001s. Calculated metadata of 3 files. 2024/10/09 14:45:14 INFO: [Cluster] [Main] SIGNAL [(15)-(SIGTERM)] received. Shutting down... 2024/10/09 14:45:14 INFO: [Cluster] [Main] Shutting down wazuh-engined (pid: 112041) 2024/10/09 14:45:14 INFO: [Cluster] [Main] Shutting down wazuh-apid (pid: 112083) 2024/10/09 14:45:14 INFO: [Cluster] [Main] Shutting down wazuh-comms-apid (pid: 112157) ```

Validate that the manager status takes wazuh-engined into account

> If it didn't, the call would fail with a message saying `wazuh-engined->stopped`. ```console root@wazuh-master:/# curl -u wazuh:wazuh -k -X POST https://localhost:55000/security/user/authenticate {"data": {"token": "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNzI4NDg2MDA5LCJleHAiOjE3Mjg0ODY5MDksInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.QecXAy-J2MEGbaNdgCrnZZiPLw87sWFsE3FqMdMvTdrpgrmP3OsTqrRXnpA3byCBysAQg-2_brOfoLipqTXHBg"}, "error": 0} ```

GGP1 commented 1 month ago

Update

Applied suggested changes.

wazuh / wazuh