search

wazuh / wazuh

Wazuh - The Open Source Security Platform. Unified XDR and SIEM protection for endpoints and cloud workloads.
https://wazuh.com/
Other
11.11k stars 1.68k forks source link

Wazuh stops when an invalid Syscollector configuration is applied #18968

Closed mauromalara closed 1 year ago

mauromalara commented 1 year ago
Wazuh version Component Install type Install method Platform
4.6.0-40600 Syscollector Agent Packages Windows-10-10.0.14393-SP0

While running the integration tests for 4.6.0-alpha1 we found that when using an invalid Syscollector configuration Wazuh does not start correctly. So, the tests fail because they expect some logs to appear, but they never do.

Evidence: syscollector-agent-windows.zip

Note: The test_syscollector_scanning test fails because of a known issue: https://github.com/wazuh/wazuh-qa/issues/4230

Error preview:

2023/09/11 12:45:21 wazuh-agent[3664] logging_helper.c:53 at loggingFunction(): CRITICAL: Error, id: dbEngine: Error deleting old db.
2023/09/11 12:45:21 wazuh-agent[3664] win_service.c:262 at OssecServiceCtrlHandler(): INFO: Received exit signal. Starting exit process.
2023/09/11 12:45:21 wazuh-agent[3664] win_service.c:268 at OssecServiceCtrlHandler(): INFO: Set pending exit signal.
2023/09/11 12:45:21 wazuh-agent[3664] win_service.c:278 at OssecServiceCtrlHandler(): INFO: Exit completed successfully.
2023/09/11 12:45:21 wazuh-agent[3664] cryptography.c:541 at verify_hash_and_pe_signature(): DEBUG: PE signature verification succeeded for C:\Windows\system32\mswsock.dll
2023/09/11 12:45:21 wazuh-agent[3664] dll_load_notify.c:101 at dll_notification(): DEBUG: The file 'C:\Windows\system32\mswsock.dll' is signed and its signature is valid.
mauromalara commented 1 year ago

Seems to be related with: https://github.com/wazuh/wazuh/issues/16080

Dwordcito commented 1 year ago

The issue is not reproducible in our environments, please help @GabrielEValenzuela to reproduce them.

GabrielEValenzuela commented 1 year ago

Hi @mauromalara !

Could you send me the steps that you used to reproduce this issue ?

I was writing this report about this.

Analysis report

Description

During the testing phase of version 4.6.0.alpha1, it has been observed that a subset of test cases is failing affecting the syscollector modules. In total, 5 out of 26 test cases have failed during this phase.

The following table provides a summary of the test cases that have failed along with the specific configuration values that trigger these failures:

Test case Configuration provided that make the test fail
empty_value_ports <ports all="no" />
empty_value_all_ports <ports all="">yes</ports>
empty_value_processes <processes />
invalid_value_disabled <disabled>invalid</disabled>
check_syscollector_scan N/A

Root Cause Analysis

Investigation

Using a manager version 4.x and the Windows agent downloaded from the packages-dev repository, we installed agent version 4.6.0 on a Windows 10 Pro build 19045 VM with the following configuration common to all manual test cases.

internal_options.conf ```text # Debug options. # Debug 0 -> no debug # Debug 1 -> first level of debug # Debug 2 -> full debugging # Windows debug (used by the Windows agent) windows.debug=2 # Syscheck (local, server and Unix agent) syscheck.debug=0 # Remoted (server debug) remoted.debug=0 # Analysisd (server or local) analysisd.debug=0 # Auth daemon debug (server) authd.debug=0 # Exec daemon debug (server, local or Unix agent) execd.debug=0 # Monitor daemon debug (server, local or Unix agent) monitord.debug=0 # Log collector (server, local or Unix agent) logcollector.debug=0 # Integrator daemon debug (server, local or Unix agent) integrator.debug=0 # Unix agentd agent.debug=0 # Wazuh DB debug level wazuh_db.debug=0 wazuh_modules.debug=0 # Wazuh Cluster debug level wazuh_clusterd.debug=0 # EOF ```

Root Cause

For each test, we provide the extract of the ossec.conf and ossec.log highlighting the differences between the expected and obtained results.

Test case: empty_value_ports #### ossec.conf ```xml no 1h yes yes yes yes yes yes yes 10 ``` #### ossec.log ```text 2023/09/12 18:12:24 wazuh-agent[8180] cryptography.c:541 at verify_hash_and_pe_signature(): DEBUG: PE signature verification succeeded for C:\Windows\system32\mswsock.dll 2023/09/12 18:12:24 wazuh-agent[8180] dll_load_notify.c:101 at dll_notification(): DEBUG: The file 'C:\Windows\system32\mswsock.dll' is signed and its signature is valid. 2023/09/12 18:12:24 wazuh-agent[8180] start_agent.c:365 at agent_handshake_to_server(): INFO: (4102): Connected to the server ([192.168.0.111]:1514/tcp). 2023/09/12 18:12:24 wazuh-agent[8180] wmodules-syscollector.c:212 at wm_syscollector_read(): ERROR: Invalid content for tag 'ports' at module 'syscollector'. 2023/09/12 18:12:24 wazuh-agent[8180] config.c:481 at PrintErrorAcordingToModules(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/12 18:12:24 wazuh-agent[8180] win_utils.c:279 at local_start(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/12 18:12:24 wazuh-agent[8180] win_service.c:262 at OssecServiceCtrlHandler(): INFO: Received exit signal. Starting exit process. 2023/09/12 18:12:24 wazuh-agent[8180] win_service.c:268 at OssecServiceCtrlHandler(): INFO: Set pending exit signal. ``` | Test result | Message | |-----------------|---------------| |Expected| ERROR: Invalid content for tag 'ports' at module 'syscollector'.| |Got| ERROR: Invalid content for tag 'ports' at module 'syscollector'.|
Test case: empty_value_all_ports #### ossec.conf ```xml no 1h yes yes yes yes yes yes yes yes 10 ``` #### ossec.log ```text 2023/09/13 12:05:28 wazuh-agent[6000] wmodules-syscollector.c:199 at wm_syscollector_read(): ERROR: Invalid content for attribute 'all' at module 'syscollector'. 2023/09/13 12:05:28 wazuh-agent[6000] config.c:481 at PrintErrorAcordingToModules(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/13 12:05:28 wazuh-agent[6000] win_utils.c:279 at local_start(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/13 12:05:28 wazuh-agent[6000] win_service.c:262 at OssecServiceCtrlHandler(): INFO: Received exit signal. Starting exit process. 2023/09/13 12:05:28 wazuh-agent[6000] win_service.c:268 at OssecServiceCtrlHandler(): INFO: Set pending exit signal. 2023/09/13 12:05:28 wazuh-modulesd:syscollector[6000] wm_syscollector.c:210 at wm_sys_stop(): INFO: Stop received for Syscollector. ``` | Test result | Message | |-----------------|---------------| |Expected| ERROR: Invalid content for attribute 'all' at module 'syscollector'.| |Got| ERROR: Invalid content for attribute 'all' at module 'syscollector'.|
Test case: empty_value_processes #### ossec.conf ```xml no 1h yes yes yes yes yes yes yes yes 10 ``` #### ossec.log ```text 2023/09/13 12:08:46 wazuh-agent[2472] wmodules-syscollector.c:188 at wm_syscollector_read(): ERROR: Invalid content for tag 'processes' at module 'syscollector'. 2023/09/13 12:08:46 wazuh-agent[2472] config.c:481 at PrintErrorAcordingToModules(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/13 12:08:46 wazuh-agent[2472] win_utils.c:279 at local_start(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/13 12:08:46 wazuh-agent[2472] win_service.c:262 at OssecServiceCtrlHandler(): INFO: Received exit signal. Starting exit process. 2023/09/13 12:08:46 wazuh-agent[2472] win_service.c:268 at OssecServiceCtrlHandler(): INFO: Set pending exit signal. ``` | Test result | Message | |-----------------|---------------| |Expected| ERROR: Invalid content for tag 'processes' at module 'syscollector'.| |Got| ERROR: Invalid content for tag 'processes' at module 'syscollector'.|
Test case: invalid_value_disabled #### ossec.conf ```xml invalid 1h yes yes yes yes yes yes yes yes 10 ``` #### ossec.log ```text 2023/09/13 12:37:35 wazuh-agent[9428] wmodules-syscollector.c:130 at wm_syscollector_read(): ERROR: Invalid content for tag 'disabled' at module 'syscollector'. 2023/09/13 12:37:35 wazuh-agent[9428] config.c:481 at PrintErrorAcordingToModules(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/13 12:37:35 wazuh-agent[9428] win_utils.c:279 at local_start(): ERROR: (1202): Configuration error at 'ossec.conf'. 2023/09/13 12:37:35 wazuh-agent[9428] win_service.c:262 at OssecServiceCtrlHandler(): INFO: Received exit signal. Starting exit process. 2023/09/13 12:37:35 wazuh-agent[9428] win_service.c:268 at OssecServiceCtrlHandler(): INFO: Set pending exit signal. 2023/09/13 12:37:35 wazuh-modulesd:syscollector[9428] wm_syscollector.c:210 at wm_sys_stop(): INFO: Stop received for Syscollector. ``` | Test result | Message | |-----------------|---------------| |Expected| ERROR: Invalid content for tag 'disabled' at module 'syscollector'.| |Got| ERROR: Invalid content for tag 'disabled' at module 'syscollector'.|
Test case: check_syscollector_scan #### ossec.conf ```xml
172.31.6.100
1514 tcp aes 10 99999999999 no no 5000 500 Application eventchannel Security eventchannel Event/System[EventID != 5145 and EventID != 5156 and EventID != 5447 and EventID != 4656 and EventID != 4658 and EventID != 4663 and EventID != 4660 and EventID != 4670 and EventID != 4690 and EventID != 4703 and EventID != 4907 and EventID != 5152 and EventID != 5157] System eventchannel active-response\active-responses.log syslog yes no yes no 1h yes yes yes yes yes yes yes yes 10 yes 1800 1d yes \\server\jre\bin\java.exe C:\cis-cat yes yes C:\Program Files\osquery\osqueryd C:\Program Files\osquery\log\osqueryd.results.log C:\Program Files\osquery\osquery.conf yes yes wpk_root.pem yes plain yes ``` #### ossec.log ```text [To big log] 2023/09/13 12:38:37 wazuh-modulesd:syscollector[8396] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending hotfixes scan ``` | Test result | Message | |-----------------|---------------| |Expected| DEBUG: Ending hotfixes scan| |Got| DEBUG: Ending hotfixes scan|

Conclusion

The issue is not reproducible in our environments.

mauromalara commented 1 year ago

Update

mauromalara commented 1 year ago

Conclusion

It's a race condition produced by the tests. 🔴

Context/background:

Manual tests

After running manual tests, the error cannot be reproduced, either by me or the developer.

Automated tests execution

I executed the tests and now the error appears.

So, we can conclude that the error appears only when automatic tests are executed.

Execution details
- Manager system info ```shell # cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy # df -H --total Filesystem Size Used Avail Use% Mounted on /dev/root 32G 2.9G 29G 10% / tmpfs 2.1G 82k 2.1G 1% /dev/shm tmpfs 813M 840k 812M 1% /run tmpfs 5.3M 0 5.3M 0% /run/lock /dev/nvme0n1p15 110M 6.4M 104M 6% /boot/efi tmpfs 407M 4.1k 407M 1% /run/user/0 total 35G 2.9G 32G 9% - # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: AuthenticAMD Model name: AMD EPYC 7571 CPU family: 23 Model: 1 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1 Stepping: 2 BogoMIPS: 4399.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonsto p_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalig nsse 3dnowprefetch topoext vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt nrip_save Virtualization features: Hypervisor vendor: KVM Virtualization type: full Caches (sum of all): L1d: 32 KiB (1 instance) L1i: 64 KiB (1 instance) L2: 512 KiB (1 instance) L3: 8 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0,1 Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Mitigation; untrained return thunk; SMT vulnerable Spec store bypass: Vulnerable Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Srbds: Not affected Tsx async abort: Not affected ``` - Manager version ``` # /var/ossec/bin/wazuh-control info WAZUH_VERSION="v4.6.0" WAZUH_REVISION="40600" WAZUH_TYPE="server" ``` - Windows Agent version ![Screenshot 2023-09-15 at 15 48 09](https://github.com/wazuh/wazuh/assets/39094716/31fec2a5-5937-4044-b2b6-d3132b50fced) - Windows Agent initial config ```xml
MANAGER_IP
1514 tcp aes 10 60 yes no 5000 500 Application eventchannel Security eventchannel Event/System[EventID != 5145 and EventID != 5156 and EventID != 5447 and EventID != 4656 and EventID != 4658 and EventID != 4663 and EventID != 4660 and EventID != 4670 and EventID != 4690 and EventID != 4703 and EventID != 4907 and EventID != 5152 and EventID != 5157] System eventchannel active-response\active-responses.log syslog no ./shared/win_applications_rcl.txt ./shared/win_malware_rcl.txt yes yes 12h yes no 43200 %WINDIR% %WINDIR%\SysNative %WINDIR%\SysNative\drivers\etc %WINDIR%\SysNative\wbem %WINDIR%\SysNative\WindowsPowerShell\v1.0 %WINDIR%\SysNative %WINDIR%\System32 %WINDIR%\System32\drivers\etc %WINDIR%\System32\wbem %WINDIR%\System32\WindowsPowerShell\v1.0 %WINDIR%\System32 %PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup %PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup\desktop.ini .log$|.htm$|.jpg$|.png$|.chm$|.pnf$|.evtx$ HKEY_LOCAL_MACHINE\Software\Classes\batfile HKEY_LOCAL_MACHINE\Software\Classes\cmdfile HKEY_LOCAL_MACHINE\Software\Classes\comfile HKEY_LOCAL_MACHINE\Software\Classes\exefile HKEY_LOCAL_MACHINE\Software\Classes\piffile HKEY_LOCAL_MACHINE\Software\Classes\AllFilesystemObjects HKEY_LOCAL_MACHINE\Software\Classes\Directory HKEY_LOCAL_MACHINE\Software\Classes\Folder HKEY_LOCAL_MACHINE\Software\Classes\Protocols HKEY_LOCAL_MACHINE\Software\Policies HKEY_LOCAL_MACHINE\Security HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurePipeServers\winreg HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\URL HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components HKEY_LOCAL_MACHINE\Security\Policy\Secrets HKEY_LOCAL_MACHINE\Security\SAM\Domains\Account\Users \Enum$ HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\AppCs HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\DHCP HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSIn HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSOut HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\RPC-EPMap HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\Teredo HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Parameters\Cache HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ADOVMPPackage\Final 60 10 100 yes 5m 10 no 1h yes yes yes yes yes yes yes 10 yes 1800 1d yes \\server\jre\bin\java.exe C:\cis-cat yes yes C:\Program Files\osquery\osqueryd C:\Program Files\osquery\log\osqueryd.results.log C:\Program Files\osquery\osquery.conf yes no wpk_root.pem yes plain ``` - Windows Agent: Internal configuration ``` # local_internal_options.conf # # This file should be handled with care. It contains # run time modifications that can affect the use # of OSSEC. Only change it if you know what you # are doing. Look first at ossec.conf # for most of the things you want to change. # # This file will not be overwritten during upgrades # but will be removed when the agent is un-installed. windows.debug=2 ``` - Agents connected to the Manager ``` # /var/ossec/bin/agent_control -l Wazuh agent_control. List of available agents: ID: 000, Name: MANAGER_IP (server), IP: 127.0.0.1, Active/Local ID: 001, Name: NAME, IP: any, Active List of agentless devices: ``` - Path from where the tests are being executed (Windows agent) ``` $ pwd /c/Users/Administrator/wazuh-qa ``` - Running the tests (stopping at the first failure with `-x` and enable debugging with `--pdb` ```python $ python -m pytest -vvv -x tests/integration/test_syscollector/ --html="report-1.html" --pdb tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_deactivation[check_syscollector_deactivation] PASSED [ 3%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_all_scans_disabled[all_scans_disabled] PASSED [ 7%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_disabled] PASSED [ 11%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_interval] PASSED [ 14%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_scan_on_start] PASSED [ 18%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_hardware] PASSED [ 22%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_os] PASSED [ 25%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_network] PASSED [ 29%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_packages] PASSED [ 33%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_ports] PASSED [ 37%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_all_ports] PASSED [ 40%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_processes] PASSED [ 44%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_disabled] PASSED [ 48%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_interval] PASSED [ 51%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_scan_on_start] PASSED [ 55%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_hardware] PASSED [ 59%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_os] PASSED [ 62%] tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_network] FAILED [ 66%] . . . > evm.check_tag_error(file_monitor=file_monitor, field=field) test_syscollector\test_syscollector_configuration.py:300: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ C:\Python37\lib\site-packages\wazuh_testing-4.6.0-py3.7.egg\wazuh_testing\modules\syscollector\event_monitor.py:299: in check_tag_error file_monitor=file_monitor) C:\Python37\lib\site-packages\wazuh_testing-4.6.0-py3.7.egg\wazuh_testing\event_monitor.py:41: in check_event callback=make_callback(callback, prefix), error_message=error_message) C:\Python37\lib\site-packages\wazuh_testing-4.6.0-py3.7.egg\wazuh_testing\tools\monitoring.py:204: in start error_message=error_message).result() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = timeout = 5 callback = . at 0x000001969E0053A8> accum_results = 1, update_position = True, timeout_extra = 0 error_message = "Could not find this event in None: ERROR: Invalid content for tag 'network' at module 'syscollector'." def start(self, timeout=-1, callback=_callback_default, accum_results=1, update_position=True, timeout_extra=0, error_message=''): """Start the queue monitoring until the stop method is called.""" if not self._continue: self._continue = True self._abort = False result = None while self._continue: if self._abort: self.stop() if error_message: logger.error(error_message) logger.error(f"Results accumulated: " f"{len(result) if isinstance(result, list) else 0}") logger.error(f"Results expected: {accum_results}") > raise TimeoutError(error_message) E TimeoutError: Could not find this event in None: ERROR: Invalid content for tag 'network' at module 'syscollector'. C:\Python37\lib\site-packages\wazuh_testing-4.6.0-py3.7.egg\wazuh_testing\tools\monitoring.py:470: TimeoutError ```

Proposed fix

This is the fix proposed for the test:

# tests/integration/test_syscollector/test_syscollector_configuration.py
.
.
.
import os
import sys
import time
.
.
.
def test_syscollector_invalid_configurations(...):
    .
    .
    .
    # Wait 100 ms to avoid race condition
    time.sleep(0.1)

After executing the tests with said fix they've passed successfully:

$ python -m pytest -vvv -x tests/integration/test_syscollector/ --html="report-1.html" --pdb -k "not test_syscollector_scanning"
============================= test session starts =============================
platform win32 -- Python 3.8.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- C:\Users\Administrator\AppData\Local\Programs\Python\Python38\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.8.0', 'Platform': 'Windows-10-10.0.20348-SP0', 'Packages': {'pytest': '6.2.2', 'pluggy': '0.13.1'}, 'Plugins': {'html': '3.1.1', 'metadata': '2.0.4', 'testinfra': '5.0.0'}}
rootdir: C:\Users\Administrator\wazuh-qa\tests\integration, configfile: pytest.ini
plugins: html-3.1.1, metadata-2.0.4, testinfra-5.0.0
collecting ... collected 27 items / 1 deselected / 26 selected

tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_deactivation[check_syscollector_deactivation] PASSED [  3%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_all_scans_disabled[all_scans_disabled] PASSED [  7%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_disabled] PASSED [ 11%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_interval] PASSED [ 15%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_scan_on_start] PASSED [ 19%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_hardware] PASSED [ 23%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_os] PASSED [ 26%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_network] PASSED [ 30%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_packages] PASSED [ 34%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_ports] PASSED [ 38%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_all_ports] PASSED [ 42%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_processes] PASSED [ 46%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_disabled] PASSED [ 50%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_interval] PASSED [ 53%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_scan_on_start] PASSED [ 57%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_hardware] PASSED [ 61%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_os] PASSED [ 65%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_network] PASSED [ 69%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_packages] PASSED [ 73%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_ports] PASSED [ 76%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_all_ports] PASSED [ 80%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_processes] PASSED [ 84%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_max_eps] PASSED [ 88%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[empty_value_hotfixes] PASSED [ 92%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_invalid_configurations[invalid_value_hotfixes] PASSED [ 96%]
tests\integration\test_syscollector\test_syscollector_configuration.py::test_syscollector_default_values[check_default_values] XFAIL [100%]

-- generated html file: file://C:\Users\Administrator\wazuh-qa\report-1.html --
=========== 25 passed, 1 deselected, 1 xfailed in 232.47s (0:03:52) ===========

More rounds are being executed to validate the change.

Update

After 3 executions it seems that the fix is not stable. The issue will be marked as qa_known.

Dwordcito commented 1 year ago

It is not a product issue according to the creator of the issue.

This type of failure is because the database file is taken by another process. If you notice this problem under unwanted behavior, please reopen it.

GabrielEValenzuela commented 1 year ago

Update Report

We found a possible relation with a previous issue that shows the same behavior in the FIM database. Issue #4692 and its corresponding pull request #4708 introduce a mechanism to retry the deletion of the database file due to a failure with the host operating system, which does not release the owner of the file immediately and causes the error of the std::remove function. We tested this behavior in our environment and we're working in a fix.