Migrate Syscollector module to new agent

[cborla]

Parent issue:

• https://github.com/wazuh/wazuh-agent/issues/15

Description

Migrate Syscollector and DBsync code from the wazuh/wazuh repository to the wazuh/wazuh-agent repository.

Tasks

1. Identify the code in the wazuh/wazuh repository.
2. Migrate the code to the wazuh/wazuh-agent repository.
3. Refactor the code as necessary to fit the new repository structure.
4. Test the migrated code to ensure it works correctly in the new repository.

Implementation Constraints

1. Code Migration: The code for Syscollector and DBsync must be migrated to the new repository. This task depends on the migration spike: https://github.com/wazuh/wazuh/issues/24037.
2. Modular Implementation: The new Inventory will be implemented as a module of the agent, following the predefined scaffolding: #1.
3. Queue Integration: Messages generated by Syscollector must be inserted into the new Queue component. #16

Dependencies

Migration spike: https://github.com/wazuh/wazuh/issues/24037

Subtasks

Initial tasks

• [x] Create the "Pool" class for modules based on concepts and implement the "Inventory" module.
• [x] Migrate the Syscollector code to the wazuh-agent repository.
• [x] Rename files, classes, functions, and variables from "syscollector" to "inventory."
• [x] Adapt CMake files to the new repository structure.
• [x] Add vcpkg to the project.
• [x] Add dependencies supported by vcpkg.
• [x] Migrate dependencies not supported by vcpkg: libdb, pacman, procps, rpm.
• [x] Migrate modules used by inventory:
  • data-provider
  • dbsync
• [x] Remove rsync.
• [x] Migrate common modules (src/common/) used by inventory.
• [x] Remove wm_syscollector.c module definition code based on struct wm_context.
• [x] Adapt "inventory" methods to the new Modules class.
• [x] Initialize and run the module.

2. [x] Investigate how to add missing dependencies to vcpkg:

   • Research how to add libdb, pacman, procps, and rpm to vcpkg.
   • This is a high priority task, as we currently need to manually copy the binaries from the Wazuh Agent 4.x build to 5.0.
   • Investigate issue #65.
   • This task is complex because it uses make instead of cmake. We may need help from Octa or the team who originally worked on it.

3. [x] Integrate with the Queue:

   • Use Task Manager to integrate the new client with the Queue component.
   • Note: This task is on hold until #82 is merged into master.
   4. [x] Testing:
      • [x] Define and create tests for the module manager:
      • [x] Migrate tests for inventory, data-provider, and dbsync.
      • [x] Create tests for the new module manager (pool.cpp) and decide if this will be the final name.

4. [x] Compile Windows with MSVC

   • [x] Adapt the agent code for windows to compile with MSVC.

5. [x] Replace to new Logger

   • [x] Adapt the agent code use the new Logger.

6. [ ] Clean up code for unsupported operating systems.

7. [x] Analyze if http-request (urlrequest) used by data-provider can be removed.

Nice to have

1. [ ] Replace unwanted libraries
   • [ ] #152
   • [ ] #155
   • [ ] #156

[ncvicchi]

Update 2024/08/20

Inventory migration

• Fixed librpm library compilation
• Fixed libprops library compilation

[nbertoldo]

Update 2024/08/20

Testing:

• Migrating inventory unit tests: _inventoryImptest, _invNormalizertest
• Update Inventory class for receiving a configuration structure.

[ncvicchi]

Update 2024/08/21

Inventory migration

• Fixed rpm compilation
• Added libdb compilation
• Succesfully building wazuh-agent

[nbertoldo]

Update 2024/08/21

Testing:

• Migrating inventory unit tests: _inventoryImptest, _invNormalizertest
• Update Inventory class.

[nbertoldo]

Update 2024/08/22

Migrate inventory unit test #102:

• Migrate _invNormalizertest
• Rename class Pool to ModulesManager and reorganize related files in /include and /src folder.

[ncvicchi]

Update 2024/08/22

• Merged Fix support for libraries not supported by vcpkg
• Researching vpckg to including the libraries as pre-compiled binaries

[nbertoldo]

Update 2024/08/23

Migrate inventory unit test #102:

• Add ModulesManager unit tests.

[nbertoldo]

Update 2024/08/26

Migrate inventory unit test #102:

• Fixed ModulesManager unit tests.
• Added extra tests.
• Rebase to the base branch.

[ncvicchi]

Update 2024/08/26

Commit c213c35

• Rebased to master
• Changed ifdefs for pragmas
• Changed hs for hpps
• Created modules library
• Moved modules execution from its own process to agent's task manager

[nbertoldo]

Update 2024/08/27

Migrate inventory unit test #102:

• Rebased to base branch and merged.
• Started integration with the taskManager.

[ncvicchi]

Update 2024/08/27

Migrate inventory unit test #102:

• Rebased to base branch and merged.
• Started integration with the taskManager.

[nbertoldo]

Update 2024/08/28

• Integration with the taskManager.

[ncvicchi]

Update 2024/08/28

• Integration with the taskManager.

[nbertoldo]

Update 2024/08/29

• Fixed segmentation fault in dbsync caused by SQliteCpp and unofficial::sqlite3 namespace conflict.
• Analyze ModuleManager integration with MultiTypeQueue and TaskManager.

[ncvicchi]

Update 2024/08/29

• Fixed segmentation fault in dbsync caused by SQliteCpp and unofficial::sqlite3 namespace conflict.
• Analyze ModuleManager integration with MultiTypeQueue and TaskManager.

[nbertoldo]

Update 2024/08/30

• Add moduleManager to the taskManager.
• Integrate MultiTypeQueue to the moduleManager and push inventory delta events.

[ncvicchi]

Update 2024/08/30

• Add moduleManager to the taskManager.
• Integrate MultiTypeQueue to the moduleManager and push inventory delta events.

[nbertoldo]

Update 2024/09/02

• Removed unused code.

• 45 ready for review.

• Inventory E2E testing using the server mock. An “Invalid request format” error occurs:

Server output

```console ==================== REQUEST START ==================== Timestamp: Mon Sep 2 17:47:01 2024 Client IP: 127.0.0.1:55884 HTTP Request: POST /stateless HTTP/1.1 Host: localhost User-Agent: Boost.Beast/353 Accept: application/json Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXUyJ9.eyJleHAiOjE3MjUzMTAwNTEsImlhdCI6MTcyNTMxMDAyMSwiaXNzIjoic29tZS1hdXRoLXNlcnZlciIsInV1aWQiOiJkNTJhYTA1Yy1jMTBkLTRmMzQtYjZhNi05NDk1OWU5NTQxMGMifQ.aIoXm4XBlKnAxC_uMFjifsJO5lBN1WqlZoCuTT_4LoI Content-Type: application/json Content-Length: 344 {"events":[[{"data":{"data":{"board_serial":" ","checksum":"51ee4bfdf4dd6179f792c57b94339053067cd2e7","cpu_cores":16,"cpu_mhz":4846.0,"cpu_name":"AMD Ryzen 7 5800X 8-Core Processor","ram_free":23144424,"ram_total":32799344,"ram_usage":30,"scan_time":"2024/08/30 21:22:31"},"operation":"INSERTED","type":"dbsync_hwinfo"},"module":"inventory"}]]} ==================== REQUEST END ====================== ==================== RESPONSE START =================== Timestamp: Mon Sep 2 17:47:01 2024 Client IP: 127.0.0.1:55884 HTTP Response: HTTP/1.1 400 Bad Request Server: Boost.Beast Content-Type: json Content-Length: 22 Invalid request format ==================== RESPONSE END ===================== ```

[nbertoldo]

Update 2024/09/03

• Rebase to master.
• Fixed error building libdb.
• Fixing compilation for macOS.

[ncvicchi]

Update 2024/09/03

• Deploying OSs VMs for testing.
• Fixed error building libdb.
• Fixing compilation for Windows.

[nbertoldo]

Update 2024/09/04

• Fix compilation for macOS.

[nbertoldo]

Update 2024/09/05

• Fixing compilation for _macOS.
• Adapting CMakeFiles.txt to download external dependencies unsupported by vcpkg from Wazuh package repository.

[nbertoldo]

Update 2024/09/06

• Add external dependencies unsupported by vcpkg from Wazuh package repository.

[nbertoldo]

Update 2024/09/09

• Fixing compilation for macOS.

[nbertoldo]

Update 2024/09/10

• Fix compilation for macOS.

[nbertoldo]

Update 2024/09/11

• Merged #115.
• Fixed agent unit tests and merged #147.
• Fixing clang-format checks #148.

[nbertoldo]

Update 2024/09/12

• Fix clang-format checks and compilation error in http-request #148.

[nbertoldo]

Update 2024/09/13

• Resolving some suggestions from the PR review: #148.

[nbertoldo]

Update 2024/09/16

• Improvements and corrections from the PR review: #148.

[nbertoldo]

Update 2024/09/17

• Remove unsuported OS:
  • Solaris
  • AIX
  • HP-UX
  • FreeBSD
  • OpenBSD
  • Alpine Linux
  • Arch Linux

[nbertoldo]

Update 2024/09/18

• Add wazuh-http-request as a Git submodule: #148.

[nbertoldo]

Update 2024/09/19

• Integrate ConfigurationParser to moduleManager: #148.

[nbertoldo]

Update 2024/09/20

• Fix some warnings treated as errors

[nbertoldo]

Update 2024/09/23

• Rebase enhancement/17-improvements-suggested-in-the-review to enhancement/17-migrate-syscollector-to-inventory-at-new-agent
• Solve rebase conflicts
• Rebase enhancement/17-migrate-syscollector-to-inventory-at-new-agent to master

[nbertoldo]

Update 2024/09/24

• Rebase enhancement/17-migrate-syscollector-to-inventory-at-new-agent to [master].(https://github.com/wazuh/wazuh-agent/tree/master)
• Remove unused libraries: http-request, _osregex, _osnet, _osxml.
• Update package versions in vcpkg.json.
• Fix rpm port compilation.

[nbertoldo]

Update 2024/09/26

• Fixed inventory unit tests.
• Removed external dependencies (src/external).

[nbertoldo]

Update 2024/09/27

• Fixing errors linking librpm.

[nbertoldo]

Update 2024/09/30

• Remove external dependencies from src/external.
• Fix error building for macOS.
• Rebase

[nbertoldo]

Update 2024/10/01

• Enable data-provider and dbsync unit tests.
• Rebase to master.