Add a module to test the syscollector configurations

mauromalara commented 1 year ago

Description

Add a test to check if the different combinations of configurations produce the desired results.

Test cases analysis & design

Analysis

> **Note**: Invalid values include empty value

		Input values
Conditions	disabled	yes	no	no	no	no tag	invalid
	interval	5s	2s	1s	5s	no tag	invalid
	scan_on_start	yes	no	yes	yes	no tag	invalid
	hardware	yes	no	yes	no	no tag	invalid
	os	yes	no	yes	no	no tag	invalid
	network	yes	no	yes	no	no tag	invalid
	packages	yes	no	yes	no	no tag	invalid
	ports	yes	no	yes	yes	no tag	invalid
	ports-> all	no	no	no	yes	-	invalid
	processes	yes	no	yes	no	no tag	invalid
	hotfixes	yes	no	yes	yes	no tag	invalid
	synchronization	yes	no	yes	yes	no tag	-
	max_eps	10	-	0	10	no tag	invalid
Expected results	Syscollector is disabled	X
	Start scanning after X seconds		X	X	X	X
	Start scanning on startup			X	X	X
	Do not store any information		X
	Scan hardware			X		X
	Scan the os info			X		X
	Scan network			X		X
	Scan packages			X		X
	Scan listening ports			X		X
	Scan all ports				X
	Scan processes			X		X
	Scan hotfixes			X		X
	Do not synchronize anything		X	X		X
	Synchronize the agent database with the manager's database				X
	X synchronization events per second are sending				X
	Syscollector does not start and show errors						X
	Default value is configured					X

Design

- Verify that the module does not start when it's disabled - Verify that the scan starts after N seconds but no scan is triggered - Verify that the scan starts on startup and collects information, but no sync is performed and the scan starts again after 1 second (scanning only listening ports) - Verify that the scan starts on startup scanning all ports and synchronizing the DB at 10 EPS - Verify that the module doesn't start and show errors when invalid or empty options are configured (1 case per tag)

mauromalara commented 1 year ago

Update 03/11/2022

Worked on component test structure

mauromalara commented 1 year ago

Update 04/11/2022

Test moved to integration suite due to syscollector dependency from wazuh-db
Finished test structure
Multiples changes to the existing framework were made

mauromalara commented 1 year ago

Update 10/11/2022

I talked with a teammate to solve a problem with the scans
I found the solution to the uninformative message problem when using pytest.raises
I made some manual testing and fixed different errors
I changed the structure of the test that checks the invalid inputs

mauromalara commented 1 year ago

Update 14/11/2022

I added support in Jenkins to run the tests automatically
The tests execution failed, so I was fixing them

mauromalara commented 1 year ago

Update 15/11/2022

Solved multiples errors
Had problems while running tests in Windows:

PS C:\Users\vagrant\Downloads> net stop WazuhSvc
System error 109 has occurred.

The pipe has been ended.

Faulting application name: wazuh-agent.exe, version: 0.0.0.0, time stamp: 0x6373ef09
Faulting module name: KERNELBASE.dll, version: 10.0.17763.3650, time stamp: 0xa9404300
Exception code: 0xc00000fd
Fault offset: 0x0011bbe5
Faulting process id: 0x13d4
Faulting application start time: 0x01d8f9308838e236
Faulting application path: C:\Program Files (x86)\ossec-agent\wazuh-agent.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 144595fb-3a81-4d4d-8929-913dbcae5ed9
Faulting package full name: 
Faulting package-relative application ID:

Asked @jnasselle for help

mauromalara commented 1 year ago

Update 16/11/2022

Bug reported: wazuh/wazuh#15412
Bug reported: wazuh/wazuh#15413
Multiple changes made to fix tests in Windows and macOS

mauromalara commented 1 year ago

Update 18/11/2022

Debugged in Windows.

mauromalara commented 1 year ago

Update 28/11/2022

I've been fixing the Windows tests

mauromalara commented 1 year ago

Update 29/11/2022

Test with the latest change ran locally for Windows but ran in Jenkins for Ubuntu, CentOS, and macOS.

Target	R1	R2	R3	Commit
Agent (CentOS, Ubuntu, Windows, macOS)	🔴	🔴	🔴	d255282
Manager (CentOS)	🟢	🟢	🟢	d255282

Conclusion: The tests still failing on Windows due to a common error

``` =================================== ERRORS ==================================== _ ERROR at teardown of test_syscollector_invalid_configurations[empty_value_disabled] _ conftest.py:962: in daemons_handler control_service('stop') C:\Python37\lib\site-packages\wazuh_testing-4.4.0-py3.7.egg\wazuh_testing\tools\services.py:146: in control_service raise ValueError(f"Error when executing {action} in daemon {daemon}. Exit status: {result}") E ValueError: Error when executing stop in daemon None. Exit status: 2 ---------------------------- Captured stdout setup ---------------------------- The Wazuh service is starting. The Wazuh service was started successfully. ---------------------------- Captured stderr setup ---------------------------- 2022-11-29 12:29:48,573 - wazuh_testing - DEBUG - Wazuh control set to True 2022-11-29 12:29:48,573 - wazuh_testing - DEBUG - Ignore error set to True 2022-11-29 12:29:48,574 - wazuh_testing - DEBUG - Restarting wazuh using wazuh-control ----------------------------- Captured log setup ------------------------------ DEBUG wazuh_testing:conftest.py:927 Wazuh control set to True DEBUG wazuh_testing:conftest.py:931 Ignore error set to True DEBUG wazuh_testing:conftest.py:940 Restarting wazuh using wazuh-control ---------------------------- Captured stderr call ----------------------------- 2022-11-29 12:30:00,712 - wazuh_testing - ERROR - Could not find this event in C:\Program Files (x86)\ossec-agent\ossec.log: DEBUG: Starting Syscollector. 2022-11-29 12:30:00,712 - wazuh_testing - ERROR - Results accumulated: 0 2022-11-29 12:30:00,712 - wazuh_testing - ERROR - Results expected: 1 ------------------------------ Captured log call ------------------------------ ERROR wazuh_testing:monitoring.py:465 Could not find this event in C:\Program Files (x86)\ossec-agent\ossec.log: DEBUG: Starting Syscollector. ERROR wazuh_testing:monitoring.py:466 Results accumulated: 0 ERROR wazuh_testing:monitoring.py:468 Results expected: 1 -------------------------- Captured stderr teardown --------------------------- 2022-11-29 12:30:00,722 - wazuh_testing - DEBUG - Stopping wazuh using wazuh-control ---------------------------- Captured log teardown ---------------------------- DEBUG wazuh_testing:conftest.py:961 Stopping wazuh using wazuh-control _ ERROR at teardown of test_syscollector_invalid_configurations[empty_value_interval] _ conftest.py:962: in daemons_handler control_service('stop') C:\Python37\lib\site-packages\wazuh_testing-4.4.0-py3.7.egg\wazuh_testing\tools\services.py:146: in control_service raise ValueError(f"Error when executing {action} in daemon {daemon}. Exit status: {result}") E ValueError: Error when executing stop in daemon None. Exit status: 2 ```

Tests re-ran after changes

Target	R1	R2	R3	Commit
Agent (CentOS, Ubuntu, Windows, macOS)	🟡	🟡	🟡	f465ea8

Tests re-run after changes

Some tests were skipped but this is unwanted behavior, so a condition was added to the pytest marker.

Also, I change the timeout to 60 + 2 seconds. The conclusion was made after the following executions:

**AGENT ALREADY INITIALIZED AND REGISTERED IN THE MANAGER - A SYSCOLLECTOR ANALYSIS ALREADY EXISTS IN THE DB** R1: ``` 2022/11/29 19:05:16 wazuh-agent[4696] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:05:16 wazuh-modulesd:syscollector[4696] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:05:16 wazuh-modulesd:syscollector[4696] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ``` R2: ``` 2022/11/29 19:06:14 wazuh-agent[3928] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:06:14 wazuh-modulesd:syscollector[3928] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:06:14 wazuh-modulesd:syscollector[3928] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ``` R3: ``` 2022/11/29 19:08:44 wazuh-agent[3648] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:08:44 wazuh-modulesd:syscollector[3648] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:08:44 wazuh-modulesd:syscollector[3648] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ``` R4: ``` 2022/11/29 19:08:49 wazuh-agent[3536] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:08:49 wazuh-modulesd:syscollector[3536] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:08:50 wazuh-modulesd:syscollector[3536] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ``` R5: ``` 2022/11/29 19:08:54 wazuh-agent[4728] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:08:54 wazuh-modulesd:syscollector[4728] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:08:54 wazuh-modulesd:syscollector[4728] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ```
**AGENT INITIALIZED FOR THE FIRST TIME WITH SYSCOLLECTOR ENABLED, AND NOT REGISTERED IN THE MANAGER** R1: ``` 2022/11/29 19:15:22 wazuh-agent[3680] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:16:22 wazuh-modulesd:syscollector[3680] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:16:23 wazuh-modulesd:syscollector[3680] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ``` R2: ``` 2022/11/29 19:20:13 wazuh-agent[4148] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:21:13 wazuh-modulesd:syscollector[4148] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:21:13 wazuh-modulesd:syscollector[4148] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ``` R3: ``` 2022/11/29 19:22:30 wazuh-agent[1468] win_utils.c:100 at local_start(): DEBUG: Reading agent configuration. 2022/11/29 19:23:30 wazuh-modulesd:syscollector[1468] wm_syscollector.c:162 at wm_sys_main(): DEBUG: Starting Syscollector. 2022/11/29 19:23:31 wazuh-modulesd:syscollector[1468] wm_syscollector.c:101 at wm_sys_log(): DEBUG: Starting syscollector sync ```

Target	R1	R2	R3	Commit
Agent (CentOS, Ubuntu, Windows, macOS)	🟢	🟢	🟢	71d6c79

mauromalara commented 1 year ago

Update 10/12/2022

Requested changes applied
Run 3 rounds in both Manager and Agent

mauromalara commented 1 year ago

Update 14/12/2022

Fixed Windows tests
Meeting with Seyla

mauromalara commented 1 year ago

Update 10/01/2022

Changes applied
Can't run tests due to an issue with pip to install the framework

mauromalara commented 1 year ago

Update 13/01/2023

Corrections applied
Environment fixed

mauromalara commented 1 year ago

Update 2023/03/01

Multiple changes applied because the requested changes

mauromalara commented 1 year ago

Update 2023/03/01

Tested in Linux (manager) but could not test in Jenkins due to maintenance.
Could not launch all the ITs locally (report-all-its.html.zip)
Launched new tests locally 🟢 report-1.html.zip report-2.html.zip report-3.html.zip

mauromalara commented 1 year ago

Update 04/14/2023

Applied changes
Rebased
Env configured
Several tests ran

mauromalara commented 1 year ago

Update 04/18/2023

Requested changes applied

wazuh / wazuh-qa